A Categorical Treatment of Malicious Behavioral Obfuscation

International audienceThis paper studies malicious behavioral obfuscation through the use of a new abstract model for process and kernel interactions based on monoidal categories. In this model, program observations are consid-ered to be finite lists of system call invocations. In a first step, we show how malicious behaviors can be obfuscated by simulating the observa-tions of benign programs. In a second step, we show how to generate such malicious behaviors through a technique called path replaying and we extend the class of captured malwares by using some algorithmic transformations on morphisms graphical representation. In a last step, we show that all the obfuscated versions we obtained can be used to detect well-known malwares in practice

Abstraction-based Malware Analysis Using Rewriting and Model Checking

International audienceWe propose a formal approach for the detection of high-level malware behaviors. Our technique uses a rewriting-based abstraction mechanism, producing abstracted forms of program traces, independent of the program implementation. It then allows us to handle similar be- haviors in a generic way and thus to be robust with respect to variants. These behaviors, defined as combinations of patterns given in a signa- ture, are detected by model-checking on the high-level representation of the program. We work on unbounded sets of traces, which makes our technique useful not only for dynamic analysis, considering one trace at a time, but also for static analysis, considering a set of traces inferred from a control flow graph. Abstracting traces with rewriting systems on first order terms with variables allows us in particular to model dataflow and to detect information leak